DOCOHENT RESOHE 



ED 110 848 



CE 004 691 



AUTHOR 
TITLE 

REPORT NO 
PUB DATE 
NOTE 

AVAILABLE FROM 



Rostker^ Bernard 

An Evaluation-Management Information System for 

Vocational Rehabilitation. 

P-5404 

Mar 75 

46p. 

The Rand Corporation, Santa Monica, California 90406 
(3r3.00) 



EDRS PRICE 
DESCRIPTORS 



MF-$0.76 HC-$1.95 Plus Postage 
Cost Effectiveness; Evaluation Methods; Federal 
Programs; *Management Information Systems; *Models; 
National Surveys; Recordkeeping; *Systems 
Development; *Vocational Rehabilitation 



ABSTRACT 

The paper provides the basic concepts and structure 
of an evaluative-management information system that meets the needs 
of vocational rehabilitation program and project managers and allows 
inter- and intra-program and project comparisons. Cost-benefit 
analysis has not Ci) provided managers with relevant information upon 
which to base their decisions, (2) provided a systematic means for 
making tradeoffs, nor (3) helped managers develop consistent 
incentives and standards to measure internal project efficiency. 
Three major factors must be agreed upon before developing an 
evaluation system: quantifiable program objectives, benefit-cost 
framework, and a control group to be used as a standard for 
estimating gains in client income attributable to the program. Som 
problems to be considered in structuring mathematical models are 
estimating client income gain and using a national labor market 
survey as a statistical control. The mathematical relationship 
between sample survey mean, sample survey size, and the prediction of 
foregone income for single clients and aggregations of clients 
influences the prediction variance. The new design for a national 
longitudinal survey is a continuous work history survey based on 
calendar months which allows for complete recording of single 
employment, multiple employment, job search, training, and 
withdrawals from the labor force. (SAuthor/AG) 
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I. INTRODUCTION 



One of the most important functions performed by government is the 
provision of manpower and vocational rehabilitative services. Over the 
last 15 years these services have become increasingly large in scale and 
broad in scope. Moreover, the clients served and the techniques used have 
undergone continuous change. Because of these changes and those that will 
undoubtedly take place in the future, it is important that program and 
project managers are continuously able to measure the value of the various 
services provided. Such measures can become an important management tool 
and provide guidelines for decisions about how available funds should be 
allocated between services and whether the returns from a given service 
justify the resources expended on it. 

Although considerable time and energy have been spent collecting 
statistics on current programs and in performing benefit-cost studies 
of special projects^ the government does not have an effective evaluative 
program that accommodates the needs of either program or project managers. 
This paper lays out the basic concepts and structure of an evaluative- 
management information system that meets the needs of program and project 
managers and allows inter- and intra-program and project comparisons. 

Plan of the Paper 

This paper is divided into five major sections. The first section ex- 
amines the general failure of benefit-cost analysis to provide project and 
program managers with relevant information upon which to base their decisions. 
It concludes that benefit-cost evaluation has been of little value because it 
has not provided a systematic means for making inter-program or even inter- 
project tradeoffs. Similarly, it has failed to help project managers develop 
consistent incentives and standards to measure internal project efficiency. 

Federally supported vocational rehabilitation programs have been in 
existence for over 50 years. In the 1960s the traditional rehabilitation 
programs were expanded to include the mentally handicapped as well as the 
physically handicapped. In 1968 the socially and economically disadvantaged 
were also included. The main thing that distinguishes vocational rehabilita- 
tion programs from other manpower programs is their comprehensive approach 
to the problems of their client population. See: Marvin B. Sussman, ed., 
So ciology and Rehabilitation (Washington, D.C.: The American Sociological 
Association) . 
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The second section examines three major factors which have precluded 
I.-irKo-HCiile Inter-program and project comparison and on which agreement 
must be reached before an evaluation system can be developed. The first 
factor is quantifiable program objectives. While not perfect, there seems 
to be some agreement that for employment related programs the best objective 
measure is improvements in the client's income stream after receiving pro- 
gram services. This will be the measure used in this paper. 

The second factor which should be addressed before developing an evalu- 
ation system is the basic benefit-cost framework to be used. While a full 
development of such a framework is beyond the scope of this paper, it is 
clear that direct comparisons among projects which are evaluated with dif- 
ferent benefit-cost methodologies are not possible. 

The third factor addressed is the development of a control group which 
can be used as a standard to estimate the gain in client income attributable 
to the program. Several different control groups are examined, i.e., random, 
before-and-af ter , matched and sample survey-regression. For a variety of 
reasons the sample survey-regression technique is preferable when large-scale, 
continuing manpower programs are evaluated. 

Tl-te third section examines the problems of estimating client income gain 
and the use of a national labor market survey as a statistical control. In 
practice, since follow-up information is collected at fixed intervals after 
closure, the problem is to see if income differentials exist over a fairly 
short period after completion of the program. Therefore, the length of the 
appropriate comparison period, and the specification of an appropriate re- 
gression equation to estimate (predict) the earnings foregone by program 
participants is critical. 

Essential in such a procedure is a small prediction variance. If client's 
foregone income cannot be accurately predicted, then one has little information 
to evaluate the program. Consideration is therefore given in the fourth section 
to the relationship between sample survey mean, sample survev size, and the pre- 
diction of foregone income for a single client as well as an aggregation of 
clients. 

The final section suggests an alternative design for national longitudinal 
survey. The development of an evaluation-management information system re- 
quires an appropriate labor market survey to be used as the basis for the 
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statistical control group. Unfortunately, existing longitudinal surveys 
do not adequately record labor market behavior. A new continuous work 
history survey design is developed, which is based on calendar months 
and allov's for complete recording of single employment, multiple employ- 
ment — both concurrent and sequential, job search, training and withdrawals 
from the labor force. 
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11. THE FAILURE OF BENEFIT-COST ANALYSIS 

The decade of the 1960s witnessed a sharp increase in the use of bene- 
fit-cost analysis to examine questions of human capital formation and the 
evaluation of specific manpower programs. Although these efforts were 
geared to "helping the decisionmaker," they had remarkably little effect 
on actual decisions. Originally, these studies were used to assess the 
"economic worth" of human capital investment programs* In fact, early 
studies (circa 1964) were read with great interest to determine the via- 
bility of the basic concept. Unfortunately, aside from establishing the 
concept, these and subsequent efforts have been of little value to either 



ic 

Benefit-cost analysis was initially used for evaluating alternative 
water resources projects. In the 1960s, with the growth of manpower pro- 
grams under the Area Redevelopment Act and the Manpower Development and 
Training Act the techniques were extended to the area of human capital 
formation. This development was coincidental with that of the theory of 
human capital. Several classics in the development of benefit-cost anal- 
ysis as a tool of applied economics are: Otto Eckstein, Water Resource 
Development (Cambridge: Harvard University Press, 1958); John V. Krutilla 
and Otto Eckstein, Multiple Purpose River Developments (Baltimore: Johns 
Hopkins Press, 1958); Charles J. Hitch and Roland McKean, The Economics of 
Defense in the Nuclear Age (Cambridge: Harvard University Press, 1963); 
Roland McKean, Efficiency in Government Through Systems Analysis (New York: 
John Wiley and Sons, Inc., 1958). The best review of benefit-cost analysis 
is: A. R. Prest and R. Turvey, "Cost-Benefit Analysis: A Survey," Economic 
Journal, 75 (December 1965): 683-735. Some of the major works in the de- 
velopment of human capital theory are: Gary Becker, Human Capital (New York: 
Columbia University Press, 1964); William Bowen, Economic Aspects of Education 
(Princeton: Industrial Research Section, Princeton University, 1964); 
Theodore W. Schultz, The Economic Value of Education (New York: Columbia 
University Press, 1963); Theodore W. Schultz, "Investment in Human Capital," 
American Economic Review , March 1961. Also see the October 1962 "Supplement" 
to the Journal of Political Economy . There have been numerous benefit-cost 
studies of specific manpower-vocational programs, many of which have been 
published in the Journal of Human Resources and the Industrial and Labor 
Relations Review . Also see: G. G. Somers and W. D. Wood, Cost-Benefit 
Analysis of Manpower Policies (Kingston, Ontario: Industrial Research 
Centre, Queen's University, 1969). 

See the early work of Gerald Somers and Ernst W. Stromsdorf er, "A 
Benefit-Cost Analysis of Manpower Retraining" paper prepared for the meetings 
of the American Economic Association and the Industrial Research Association, 
Chicago, December 28, 1964. 
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program managers or project managers, who have to make real resource de- 
cisions. For example, at the aggregate program level, Peter Barth argues 
that traditional 

evaluation of a single program is of limited usefulness 
typically, because it does not offer a set of feasible alterna- 
tives for making tradeoffs, Aa evaluation that is limited to 
the finding that a substantial net payoff exists for a specific 
program gives the planner no guide as to the other programs that 
might be cut back .... A related issue is the magnitude or degree 
to which good or bad programs are expanded or reduced. This ques- 
tion represents an enoinnous problem for the policymaker, yet evalu- 
ators have neglected it almost entirely.* 

In addition, generalizations about large-scale programs that are based 
upon the outcome of traditional "small-scale** studies can be dangerous. 
Small-scale studies are seldom truly representative of the larger potential 
client population, nor are they indicative of the possible variations in 
project organization or diverse environmental settings. 

If benefit-cost analysis has failed the program manager, it has been 
irrelevant and even burdensome at the project level. While evaluators 
like to talk about decisionmakers, few studies attempt to understand the 
actual decisions and options faced by project managers. As a result, 
studies are frequently ill conceived and poorly timed. The information 
they do provide is available ex post and does not affect real time resources 
decisions. A recent examination of these studies by the Urban Institute 
found that it was the basic 

... design characteristics (of the traditional studies) which 
severely restrict their reliability and usefulness, (a) They 
have been one-shot, one-time efforts when we need continuous 
evaluation of program, (b) They have been carried out in terms 
of program categories and very weak on process data when we need 
analysis that gets at the underlying assumptions of these programs, 
(c) They have been small sample studies working with gross averages, 
when we need studies large enough to allow analysis of the wide 
natural variation we know exists in costs and performance among 
projects within programs.** 
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Peter S. Barth, **0n Interprogram Manpower Studies,** in Evaluating the 
Impact of Manpower Programs , Michael E. Borus, ed. (Lexington, Massachusetts: 
D. C. Heath and Company, 1972, p. 4). 

John W. Scanlon, Joe N. Nay and Joseph S. Wholey, **An Evaluation System 
to Support a Decentralized, Comprehensive Manpower Program,** in Evaluating the 
Impact of Manpower Programs , Michael E, Borus, ed. (Lexington, Massachusetts: 
O . D. C. Heath and Company, 1972,Fp^ 28). 
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The failure of benefit-cost analysis to deal effectively with project 
decisions is seen by the distinction that evaluators often make between 
process and outcome evaluation. Process evaluation is defined as admin- 
istrative monitoring and outcome evaluation as benefit-cost analysis. 
Although evaluators continue to talk of the net benefit derived from 
program participation, project managers continue to base resources de- 
cisions on such process indicators as case load, training class size and 
case closures. For example, in California an employment counselor was 
judged to be performing satisfactorily if he had 30 case closures a moith. 
Similarly, Berkowitz and Anderson found that 

... in the traditional (vocational] rehabilitation project, 
the concept of rehabilitated or not rehabilitated (employed— 
not employed) is strictly dichotomous. No consideration is 
usually given to the quality of the job which the person re- 
ceives, and no comparison is made in the crude statistics 
between the job he held before entering the project and the 
one he is placed in after completing services. Thus a client 
whose training has benefited him to the extent that he has 
substantially improved his position in the labor market, is 
recorded in the same fashion as one who receives a menial job 
at perhaps less than the going rate, in the area.** 

Traditional process indicators are meaningless in terms of the needs 
of the client population, and they establish perverse incentives and be- 
havior inconsistent with program goals. 

The consequence of measuring performance in terms of process indicators 
can be illustrated by considering two employment counselors. Assume one 
places 50 clients in menial jobs and another places half that number in sub- 
stantial positions. Which has done the better job? Assume that the net 
benefit to the client population and society is greater in the second case. 
Although that should show up in the final benefit-cost analysis, the immediate 



For example, see: Glen G. Cain and Robinson G. Kollister, "Evaluating 
Manpower Programs for the Disadvantaged," in Cost-Benefit Analysis of Man - 
power Policies . G. G. Somers and W. D. Wood, eds. (Kingston, Ontario: 
Industrial Researcl Centre, Queen^s University, 1969, pp. 120-121). 

Monroe Berkowitz and Merilee Anderson, PADEC — An Evaluation of an 
Experimental Rehabilitation Project (New Brunswick: Bureau of Economic 
Research, Rutgers University, The State University of New Jersey, 1974, 
p. 31). 
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effect is to support behavior of the first counselor. If benefit-cost 
analysis established criteria against which the program and project are 
to be measured, those criteria should be reflected in the internal in- 
centive structure of the project. It makes little sense to establish 
operating incentives that do not support program objectives or to evaluate 
a program in terms of one standard when those charged with carrying out the 
program march to a different drummer. Process evaluation should be consis- 
tent with output evaluation. 

In sum, traditional benefit-cost evaluation has been of little value 
to the program manager it purports to serve, because it has not provided 
a systematic means for making inter-program or even inter-project tradeoffs. 
Similarly, it has failed to help project managers develop consistent incen- 
tives and standards to measure internal project efficiency. 
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III, PROGRAM AND PROJECT COMPARISONS 

Before large scale inter-program and project comparisons are feasible 
agreement must be reached on several major issues. 

1. Quantifiable program objectives 

2» A consistent benefit-cost framework 

3. A consistent, statistically acceptable and economical 
control group methodology. 

Program Objectives 

Any evaluation system must be based on an articulated set of goals 
that can be objectively related to acceptable output measures. Most pro- 
grams operate under specific legislative mandates, but for a variety of 
reasons the specification of goals and the determination of proxies for 
output measures are not always successful. Development of consistent goals 
and proxies across programs with vastly different orientations may not be 
possible. It may never be possible to compare Head Start with a vocational 
rehabilitative program designed to service handicapped adults. However, 
within general program areas one should be able to specify consistent goals 
and proxies, and among projects that are part of a given program there should 
be no problem in establishing consistent measures of performance. 

In the area of manpower-vocational programs there is general agreement, 
based on specific legislative mandates, that the primary goal is improvement 
in employment performance of program participants. Evaluators of manpower- 
vocational programs appear to agree that for employment related programs the 
best objective measure of program and project performance is improvements in 
the client's income stream after receiving program services. Improvements in 
the income stream result from a reduction in unemployment, an increase in job 
stability, an increase in wages received, and allows comparison across pro- 
grams and among projects that serve different client groups. This will be 
the output measure used in this paper. 



•k 

See: Cain and Hollister, op. cit ., pp. 122-125. 
See Barth, op. clt .. p. 6. 
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Even if there is agreement as to the primary goal of employment related 
programs, there is substantial disagreement on how to treat non-labor -market 
performance. This manifests itself as conflict between program managers and 
project operatives. The latter frequently argue that measures tied only to 
actual job performance fail to recognize subjective changes of "well-being" 
that are unrelated to labor market performance. V?hile some quantification 
of such factors as changes in dependency status for the handicapped or im- 
provements in antisocial criminal behavior may be possible, a full accounting 
is beyond present methodologies* Moreover, even if further work on social 
indicators eventually provides operational measures, the problem of finding 
a basis for making tradeoffs required for real decisions would remain. 

The establishment of goals and proxies for use in evaluation involves 
more than an abstract exercise forced by evaluators. It is a prime factor 
affecting management control. If process and outcome evaluation should be 
consistent, then the translation of explicit goals into incentives and stan- 
dards can greatly affect program operation. The case of the Job Agent pro- 
gram in California provides a clear example. 

In 1968, the California legislature created the Department of Human 
Resources Development as a comprehensive manpower agency with the goal of 
improving services for the disadvantaged. To help implement these goals 
the legislature created a new class of employment counselor, known as job 
agents. They were modeled after the vocational rehabilitation counselor 
and were to "develop individualized placement plans leading to continued 
self-sufficient employment for eligible clients with the most difficult 
problems of unemployment." They were to develop "innovative, new and 
original ways of achieving continued employment for clients." The Job 
Agent Program was unique in another way. The legislature mandated that: 

At such time as job performance standards have been devel- 
oped and performance measurement is feasible, the director 
[of Che Department of Human Resources Development] shall 
recommend to the State Personnel Board the establishment of 
a form of compensation for agents ... based primarily on the 
job agent's achievements in obtaining successful completion 
of training and employment goals by eligible persons.** 
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*California State Personnel Board Examination Notice, "Job Agents." 
August ''O, 1969. 

**Caiifornia Assembly Bill No. 1463, Article 3, Section 9701, p. 10. 
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In clear language the legislature defined the functions of job agents and 
directed that their performance be measured in terms of client employment. 

The Director of the Department clearly understood that performance 
standards and measures would not only facilitate the legislative mandate 
but would extend management control. Without measurable standards there 
was no way to determine performance except by the subjective judgment of 
supervisors. By stipulating measurable standards, management at all levels 
had a consistent means of evaluating the performance of individual job 
agents. Furthermore, measurable standards could be used by management to 
make inferences as to what attributes successful job agents possess, which 
agents work best with which clients, and which social services are most 
useful in bringing about employment improvements. Similarly, measurable 
standards provided the job agent with a yardstick he could use to recog- 
nize what management expects of him and to determine how well he was 
responding to their expectations. 

Management supported the establishment of goals and measurable stan- 
dards as a means of extending their control over the job agent. For exactly 
the same reason the job agents themselves resisted these standards. Manage- 
ment and the State legislature appeared to view the goals of the program as 
quite specific: to place and to keep disadvantaged clients in meaningful 
employment. Many job agents interpreted the goals of the program much more 
broadly: to provide any service—not only those directly related to em- 
ployment—agents felt would be beneficial to the disadvantaged community. 
They therefore resisted the development of specific goals or standards.* 

Benefit-Cost Framework 

Even when there is agreement that the primary goal of manpov/er-vocational 
programs is to improve the employment experience of clients, and that this can 
best be measured as a change in the client's income stream, direct comparisons 
among projects which are evaluated with different benefit-cost methodologies 
are not possible. Benefit-cost methodologies differ in the way they treat 



See; David H. Greenberg, Bernard D. Rostker and Leonard V. Scifers, 
The California Job Agent Incentive Pay Plan: A Case Study for Manpower 
Management , The Rand Corporation, P-5068, August 1973, 
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secondary benefits, spillover effects, transfer payments, opportunity 
costs and foregone earnings; in the way they price and treat nonmarket 
behavior; and in their general viewpoint. Benefit-cost calculations 
can be made from the point of view of society, the individual client, 
and the government. VThile everyone might agree that the program produced 
a given change in the income of a client, the calculation of benefits and 
costs from the different viewpoints results in different measures of per- 
formance. For example, participation in a vocational rehabilitation program 
might result in a reduction in remedial services needed by a handicapped 
client. How should this reduction be counted for purposes of program evalu- 
ation? A case can be made that since the services provided represented 
real resources they should be considered a reduction in cost — a real benefit. 
However, such services are generally considered transfer payments, and there 
is a long tradition that from the view of society transfer payments should 
not be counted in calculating benefits and costs. The individual considers 
the reduction in services a loss, although it might signify an improvement 
in dependency which he would value as a nonquantif iable benefit. The gov- 
ernment, which formerly provided the service, would surely consider a 
reduction in services demanded a benefit. 

The development of an appropriate and consistent benefit-cost model is 
beyond the scope of this paper. However, problems caused by not having such 
a framework are not abstract. For example, in conducting a recent evaluation 
of the Perth Amboy Diagnostic and Employability Center by the Rutgers Bureau 
of Economic Research, the Bureau attempted to compare the Perth Amboy project 
with two similar projects undertaken in different parts of the country. They 
noted that the first comparison study was executed from the view of society 
(the Wisconsin Study), and the second was done from the view of the govern- 
ment (the Michigan Study). The Rutgers group counted earnings foregone during 
training as a cost. The Wisconsin study did not count foregone earnings at 
all. And the Michigan group, working from the viewpoint of the agency, de- 
ducted foregone earnings from the benefit stream. Michigan was the only group 
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to count the net decrease in economic dependency as a benefit, again because 
they took the perspective of the government. 

Control Group Methodology 

Even when evaluators agree on quantifiable program objectives and an 
appropriate benefit-cost framework, they must find an acceptable way to 
estimate the gain in client income attributable to the program. In general, 
control groups are established to answer the question, "What would have been 
the client's income had he not participated in the program?" Choosing a 
control group that provides the best answer to this question is considered 
by most ev&luators their most difficult methodological problems. 

On purely statistical grounds, the most widely preferred control group 
is one formed by random assignment of potential program clients into two 
groups. The first, the treatment group, receives program services. The 
second is denied services, thereby providing the control. This method 
closely follows experimental procedures used in agricultural and medical 
research, as well as in psychological and sociological research. However, 
on a number of counts random assignment is inappropriate for the kind of 
continuous evaluation that is required by program and project managers. 

Random assignment creates major practical, legal, and moral problems.** 
To a person who needs and desires program services but is randomly, and thus 
arbitrarily, selected for the control group, such a program is the ultimate 
hoax. In fact, since the potential client is recruited, screened, and ten- 
tatively accepted—at least until the random selection is made— the mere act 
of rejection implicit in being selected for the control group could sufficiently 
affect attitudes to negate the randomness of the control group. The ultimate 
absurdity of random selection is the contention that not only should control 



Berkowitz and Anderson, op. cit ., pp. 133-138. For a further discussion 
on this point also see: Einar Hardin, "Benefit-Cost Analysis of Occupational 
Training Programs: A Comparison of Recent Studies," in Cost-Benefit Analysis 
of Manpowe r Policies . G. G. Somers and W. D. Wood, eds. (Kingston, Ontario: 
Industrial Relations Centre, Queen's University, 1969, pp. 97-118). 

For a further discussion of tUese problems, see: David A. Miller 
On the Choice of Control Groups: Comment," and Abraham Stabler, "On the 
Choice of Control Group: Comment," both in Evaluating the Impact of Ma npower 
|rosrams, M^^^ E. Bonis, ed. (Lexington, Massachusetts: D. C. Heath and" 
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assignees be prevented from receiving program services, but that in order 
to ensure the research design and "in order to create a sufficiently long- 
after treatment period of observation on both groups, he (the evaluator) 
must normally arrange to bar those randomly selected for no treatment from 
entering substitute manpower programs for a specific length of time/* 

Random assignment is inappropriate on other grounds. In practical 
terms random assignment is best administered in a small-scale study, oc- 
curring in a specific place at a specific tima. However, it is just this 
type of small-scale, one-shot study that has failed to provide the required 
information to either program managers or project managers. Random assign- 
ment may be useful in testing out initial concept, such as income maintenance, 
or in determining the general response to alternative health insurance plans, 
but it is not applicable to the evaluation of large-scale, continuing man- 
power programs. 

If random assignment is not feasible or even appropriate for the evalu- 
ation of continuing manpower programs, are there alternative methodologies 
that allow estimation of client income gains? The answer is a qualified 
yes. In general, "Statistical control in a nonrandom setting becomes 
adequate only when all important variables affecting the dependent variable 
are measured and are included in tables or equations in proper mathematical 
form."*** While alternative methodologies currently available may not 



Einar Hardin, "On the Choice of Control Groups," in Evaluating the 
Impact of Manpower Programs , Michael Borus, ed. (Lexington, Massachusetts: 
D. C. Heath and Company, 1972, p. 55). Hardin does note, "Little opposition 
is likely to arise (to random assignment) when the benefits to the individual 
are somewhat uncertain, when persons excluded from the program do not long 
remain barred, when random assignment is only among forms of treatment and 
treatment of some kind is offered everyone, or when there are more well- 
qualified applicants than program vacancies. In opposite circumstances, 
randomization may not be feasible." Unfortunately, in the evaluation of 
large-scale and continuing programs none of Professor Hardin's conditions 
are met. 

For a further discussion, see: Cain and Hollister, op. cit ., pp. 126- 

128. 

***Hardin, "On the Choice of Control Groups," op. cit ., p. 54. 
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saciafy the statistical purist, they can help meet the "rules of evidence" 
and provide considerable help in evaluating manpower programs.* 

One of the most widely used approaches to evaluate vocational 
rehabilitation programs is based on comparisons of client's income before 

icic 

and after participation in the program. In past studies this technique 
was used because it was inexpensive—no independent control group was con- 
structed—and the necessary data were readily available. Unfortunately, 
this technique is based on a set of unrealistic assumptions. For example, 
if a client was unemployed before entering the program the "before and 
after" technique assumes that he will remain unemployed; i.e., his future 
income stream, had he not participated in the program, is zero. This is 
most unlikely and provides an overestimate of program performance. 

Two other nonrandom techniques provide a better basis for establishing 
the performance of a program of given project. They are the "matched con- 
trol group" and the "sample survey control group" methods.*** To develop 
a matched control group the evaluator seeks a group of individuals having 
similar characteristics to those in the treatment or program group and com- 
pares the responses of the "treated" and the matched groups. In practice 
there are many sources from which evaluators can draw a matched control 
group. Several early studies of manpower training drew matched control 
groups from the files of the local offices of State Employment Service. 
Earl Main used a "snowball" method by having each program participant iden- 
tify a friend, neighbor, or relative who was in a similar economic situation. 
In his review of past studies, Hardin characterizes matched control groups 
as coming from the potential treatment population, from program dropouts, 
from the same high school, or from a group of qualified interested non- 

•kicicic 

enrollees. 



For a further discussion see: Cain and Hollister, op. clt.. p. 120 

and pp. 149-152. 
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This approach was used in evaluating the Perth Amboy experimental 
rehabilitation project. See: Berkowitz and Anderson, op. cit .. p. 121. 
In addition see: Ronald W. Conley, "A Benefit-Cost Analysis of the Voca- 
tional Rehabilitation Program," The Journal of Human Resources , IV, 2 
(Spring 1969), pp. 226-252. " 

For a rigorous statistical treatment of these methods, see: 
Gus Haggstrom, A Comparison of Alternative Methods for Estimating Treat- 
men t Effects . The Rand Corporation, P-5067, August 1973. 

ickick 

See Hardin, "On the Choice of Cc^tjr^ Groups, op. cit .. pp. 45-53. 
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In the survey control group, the evaluator uses sample survey tech- 
niques to collect an extensive set of socioeconomic, personal, and 
environmental data from the population at large. The exact data col- 
lected include factors that directly affect employment and labor market 
performance and that should be held constant when the effects of the pro- 
gram are being determined. Since the members of the survey control group 
are not "identical" to the members of the treatment group, the differences 
between the two groups are statistically controlled, using analysis of 
covariance or other regression techniques. 

Neither the matched control group nor the survey control group tech- 
niques can guarantee unbiased estimates of program performance. Neither 
technique can guarantee that the treatment group does not differ from the 
control group in some attribute not accounted for in the set of character- 
istics used to make the match or explicitly entered into the regression 
equation. However, the potential bias can be reduced if a set of explanatory 
variables is chosen that theory and past empirical studies have shown to be 
important determinants of labor market behavior. Although not ideal, the 
national control group may be superior to the matched control group in this 
regard because the evaluator can control for more characteristics in the 
regression equation than he is able to in making the match. 

The survey control group appears to be superior to the matched control 
group on several other grounds. First, by using survey data and the regression 
model of the labor market performance of control group members, the evaluator 



As Herbert Fames and John Shea noted, "The Longitudinal Surveys are 
admirably suited ... (as a survey control group) because they provide mea- 
sures of a substantial number of characteristics that are related to labor 
market success and that are also likely to be related to the probability of 
an individual applying for and being accepted into a manpower program ... 

"Moreover, by introducing such explanatory variables into a multi- 
variate analysis of the experience of both the experimental and control 
group, one would be able to move farther toward isolating the independent 
effects of the manpower program than most studies that have not been able 
to use random control groups." 

Herbert S. Fames and John R. Shea, "The Use of National Longitudinal 
Survey Data," in Evaluating the Impact of Manpower Frogram s» Michael E. Borus, 
ed. (Lexington, Massachusetts: D. C. Heath and Company, 1972, p. 194). 
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is able to estimate the treatment effect individually for each member of 
the treatment group—for each program participant. In theory this could 
also be done if the matched control group was made up of individual matched 
pairs, but in practice the quality of the match rarely justifies this pro- 
cedure. (Later in this paper I develop a model of "client income gain" which 
uses this feature. The ability to estimate the treatment effect for each 
individual client and the ability to aggregate these results allows inter- 
and intra-project and program evaluation.) 

Second, the survey control group is extremely economical and statis- 
tically efficient when used to evaluate large-scale, continuing manpower 
programs. The matched control group method requires each project to es- 
tablish and maintain a "local" control group, which is costly. Further- 
more, since the control group must be specific to a given project, control 
group observations from one project cannot be used in the analysis o£ another 
project. In practical terms, each locally matched control group would be 
small, especially compared to the size of local projects. However, the 
survey control group, because it uses regression techniques to control for 
differences between the control and treatment populations, can be used by 
all projects. The burden of establishing and maintaining the survey control 
group would be handled by a central, professional survey research center, 
rather than falling on local projects. Although a large sample survey would 
be costly, the costs would not be borne by any single project but would be 
spread and amortized over many projects or even programs* The single survey 
control group is also statistically efficient because all observations in the 
survey would enter into the regression analysis of labor market behavior, 
which is the basis for the statistical control. In effect, the size of the 
control group for each and every project would be increased, and, other things 
equal, it would increase the statistical reliability of the control group. 
(A full discussion of statistical validity and sample size is developed below.) 

Third, the national survey and regression analysis can explicitly take 
into account factors affecting program participation. For example, the sample 
survey might include questions about a person's motivation concerning the pro- 
gram, or about the availability of similar programs which could reduce the 
selectivity bias caused by the nonrandomness of the control group. 
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Fourth, the national control group would establish a common standard 
against which all projects within a given program could be measured and 
against which alternative programs could be evaluated. Difficulties caused 
by the lack of a common control group can be seen by Stromsdorfer 's expe- 
rience in West Virginia, In his analysis of a single project he estimated 
that the gain in client income ranged from $511 to $1310, depending on the 
choice of control group. In another study Borus found a similar range ($424 
to $1176), depending on the selection of control group. 

Fifth, the development of a national survey not only provides the basis 
for a common control group, it also provides an extremely useful data base 
for investigating labor market behavior. Such a survey would become a major 
source of empirical information that scholars could use to further our under- 
standing of the employment performance, for example, of handicapped and dis- 
advantaged persons. In turn, this added knowledge could be used by evaluators 
to develop improved statistical control groups. 



See: Hardin, "Cn the Choice of Control Groups," op. cit . , p. 47, and 
Hardin, "Benefit-Cost Analysis . . . op. cit . , p. 105. 
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IV. THE MODEL 



Given an appropriate national labor market survey, the expected behav- 
ior of program clients can be predicted from the actual behavior of statis- 
tically similar people in the survey population. This requires modeling 
the economic behavior of control group members and estimating the model 
using appropriate statistical techniques. 

Measuring Income Differentials 

In theory, the primary benefit derived from participating in the program 
is the measurable change in lifetime earnings, i.e., 

(1-r) 

= observed income in client in year t 

= expected income of client in year t, 
had he not participated in the program, 
inferred from an analysis of the behavior 
of the survey population 

d = expected years to live 

In practice, evaluators never have observations of lifetime earnings for 
program participants. Typically, follow-up information is collected at fixed 

icic 

intervals; usually six, 12 or 18 months after closure. The problem is to 
determine if there is an income differential over a fairly short period. 



where 



ERLC 



This equation, for ease of explanation, does not take into account the 
probability of death or other exogenous labor market factors. For a more com- 
plete specification see Bernard Rostker, The Economics of Manpower Retraining , 
Unpublished Master's Thesis, Syracuse University, January 1966, and Charles M. 
Grigg, Alphonse G. Holtmann, Patricia Y. Martin, Vocational Rehabilitation for 
the Disadvantaged (Lexington, Massachusetts; D. C. Heath and Company, 1970, 
p. 117). 



Borus and Buntz note some exceptions. See: Michael E. Borus and 
Charles G. Buntz, ''Problems and Issues in the Evaluation of Manpower Programs, 
Industrial and Labor Relations Review , Vol. 25, No. 2, January 1972, p. 240. 
For a discussion of follow-up strategies, see: Berkowitz and Anderson, op. cit . 
pp. 55-65. 
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Evaluators often extrapolate this difference over the client *s expected 
life. However, this assumes that the effects of the service as measured 
shortly after closure are constant and the differential does not decrease 
over time — a questionable assumption. 

In general, the effect of participation in a vocational program on the 
income of clients is the difference between predicted and observed income. 
Because programs attempt to improve the economic situation of clients by 
reducing current periods of unemployment, as well as increasing future wage 
levels and job stability, the appropriate comparison period begins when a 
client joins the program and ends after the post-closure follow-up period. 
Since the length of time a client spends in the program is variable, the 
comparison period for each client is also variable. Accordingly: 



T = Y - y 
t t ^t 



T =« income differential 

= client *s observed income in period t 

y = client's foregone income — the income 
^ a person with the same set of char- 
acteristics can expect in period t, 
based upon the behavior of the national 
survey group 

t = duration of program plus predetermined 
follow-up period 



''There is substantial evidence that earnings gain declines over time, 
''ee: G. G. Somers and G. H. McKechnie, "Vocational Retraining Programs for 
the Unemployed" in Industrial Relations Research Association, Proceedings 
of the Twentieth Winter Meetings > 1967, pp. 25-35; G. G. Somers and 
E. W. Stromsdorfer, "Benefit Cost Analysis of Manpower Retraining" in 
Industrial Relations Research Association, Proceedings of the Seventeenth 
Annual Meetings > 1965, pp. 172-185; M. E. Borus and E. C. Prescott, "The 
Effectiveness of MDTA Institutional Training Over Time and in Periods of 
High Unemployment." Paper presented to the Business and Economics Section 
of the American Statistical Association, December 1973. 



(1) 

where: 
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By comparison, evaluative studies often compare client and control 
group income only for a fixed period after closure.* In such cases only 
benefits derived from improved wage rates and job stability are recorded. 
This fails to account for the possible positive effect the program had in 
reducing unemployment . Moreover, the program can also have a negative ef- 
fect upon unemployment which must also be counted. Participation in the 
program can Lncreaae the time a person is without work by substituting the 
program for normal joh search procedures and placing the client in an ex- 

kit 

tended training program. [Presumably, training is an investment, and 
earnings foregone during training will be more than made up by increases 
In post-training inco:Be.] Therefore, the appropriate comparison is from 
when a person enters the program, not from when he is placed.*** 

The estimate of foregone earnings for ? client with a given set of 

A 

Characteristics is the predicted income y, based upon tho following re- 
gression using the appropriate national survey data. 

y = a + b^X^ + b^X^ •»• b3X3 + b^X^ + b5X5 + b^X^ 

y = future income over the next period 
= vector of demograph characteristics 

= vector of work history before the period 

= vector of local labor market factor during the period 

X^ = current duration of unemployment, if employed X. = 0 



(2) 

where: 



All before and after studies ignore the effects of the program on 
unemployment. However, studies which use more carefully constructed con- 
trol groups also make this mistake. See Doris Hull, "WIN II Longitudinal 
Impact Evaluation; Comparison Group Issues and Recommended Methodology," 
Unpublished technical note from KETRON, INC. to Stan Markuson, Office of 
Evaluation, Manpower Administration, USDL, January 16, 197^, p. 8. 

Berkowitz and Anderson note, "Many clients were resistant to partici- 
pating In work evaluation bxjcause of the financial pressures to find immediate 
♦employment. They evidenced similar reluctance when faced with the prospect 
of further training. In other words, long-range planning was clearly a 
luxury to them." Berkowitz and Anderson, op. cit ., p. 52. 

For an alternative specification see: Bernard Rostker, An Econometric 
Model for the Evalu ation of Manpower Prof> rams. The Rand Corporation, P-4944-1, 
January 1974, pp. 3-5. 
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X- = vector of factors relating to program participation 
and the availability of similar services to the 
survey population 

X, = vector of handicap factors, type and degree 
o 



Four features should be noted about the above equation. First, equa- 
tion (2) is a reduced form equation. Since we are interested only in 
predictions of future income (y) , the problem of bias in individual coeffi- 
cients can be ignored. 

Second, since the comparison between predicted and observed client 
income takes place from the point a client enters the program to some point 
in the future, the regression must also "look** towards the future. Demo- 
graphic status (Xj^) > current duration of unemployment (X^) , service avail- 
ability (X^) and handicap status (X^) all refer to a given point in time, 
presumably when the survey is taken. Work history (X^) is retrospective and 
refers to the period before the survey. Future income refers to observed 
income between the initial and subsequent surveys. Labor market factors 
(X^) refer to exogenous factors during the interval between st*rveys. 

Third, since the comparison period is variable, equaling the time be- 
tween a client joins the program and the post-closure follow-up, and since 
the surveys are conducted at fixed intervals, the prediction must be normal- 
ized as follows: 

^ = ^ • 1 
^t ^s * s 

where: y^ = client *s foregone income in period t 

y = predicted income in period S — the 
® income a person with the same set 
of characteristics can expect in 
period S, based upon the behavior 
of the national survey group 

S = interval between surveys 

t = comparison period — duration of program 
plus predetermined follow-up period 



See: E. Malinvaud, Statistical Methods of Econometrics (Chicago: 
Rand McNally and Company, 1966), p. 545. 
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Fourth, y is predicted individually for each program participant 
and the estimates of the income differential, i.e., treatment effect, is 
available for each client. Since individual clients are the basic treat- 
ment unit, it is possible to aggregate estimates of the treatment effect 
by counselor, work center, treatment technique, project or program. Fur- 
thermore, the estimates of treatment effects can be used as the dependent 
variable in a multi-variate analysis of performance. This should provide 
project and program managers with information needed to evaluate services 
at any level of detail. 



ERIC 
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V . STATISTICAL VALIDITY 



As argued ab^i^e, the use of a survey control group requires that the 
expected behavior of program clients can be predicted from the actual be- 
havior of statistically similar people in the survey population. Essential 
in such a procedure is a small prediction variance. If a client's foregone 
income cannot accurately be predicted, then one has little information to 
evaluate the program. The following section will indicate some of the 
general difficulties of prediction and suggest an already developed and 
tested model for choosing an experimental design for the sample survey that 
can minimize the variance in prediction subject to cost and other policy 
constraints. 

Variance of a Sin';^le Prediction 

Expression fDr the variance of a single prediction may be found in 

it 

almost any standard test on statistics or econometrics. For expository 
purposes, presented is the simplest of all cases — that of making a single 
prediction from a two-variable model. 
Suppose the model is: 



Y « a + 0X + n 



all the classical regression assumptions hold, and the parameters a and 
6 are estimated in the least squares fashion. Then, the estimate of Y^ 
for a given Xq is, 

^0 = " ^ ^0 



For example, see: J. Johnston, Econometric Methods (New York: McGraw- 
Hill Book Company, 1963, p, 36), and Arthur S. Goldberger, Econometric Theory 

(New York: John Wiley & Sons, Inc., 1966, pp. 169-170). 
** 

Using Johnston's notation. 
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and the estimate of the variance of the prediction of Y, Yq, for a given 



value of Xq is: 



VAR (Yq) = 



1 - 

1 + - + — y 

n n 



where is the estimated variance of the residuals, n is the sample size, 
is the ith observation of the X's in the sample and X is the sample mean. 
The above expression indicates that the variance of the prediction de- 
pends both on the sample size and the distance between the sample mean and 

In particular, for a given 



the specific value of X^ under consideration 



sample size, the further X^ is from the sample mean, the larger the variance 
of the prediction. 

The relationship between the variance of the prediction and the sample 
size is particularly important. As argued above, the survey control group 
may be relatively large. Since the regression model and the sample survey 
will be used as a basis for evaluating many projects, the cost of a large 
sample can be amortized over many users. Specifically, for a desired level 
of precision in prediction [VAR (Y^) = y] the required sample size is 



2 "7 
(1 + a"^) 
u_ 

"2 



where: for any of interest A meets the condition 

X 



Ao^ < Xq < X + Aa^ 



The above indicates that the "goodness of fit" of the regression equation to 
the survey data presents a limit on the variance of the prediction. Clearly 
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"2 

Y > cr^ because one cannot have a smaller prediction variance than the re- 
sidual variance of the regression. 

^"2 

The above also indicates that given an estimate of a and a desired 

u 

tolerance, there is always a sample size that will compensate for the 
difference between and the sample mean. 

Wiilfi t..e above is illustrative of errors in prediction and can be 
extended to a multiple regression model in an analogous fashion, it does 
not capture the full extent of the variance when one aggregates predictions 
for project and program evaluation. 



The Variance of Multiple Predictions 

In the evaluation of large scale and continuous manpower programs one 
is generally less interested Ln a single prediction for a single client, 
than in an aggregation of many predictions and clients into a management unit* 
Such a unit might be an individual counselor, work center, treatment tech- 
nique, project or program. In such a case, because a single regression is 
used to make all predictions, the individual predictions are not independent. 
Therefore, the variance of the aggregate prediction is not simply the linear 
combination of the variances of the individual predictions that make up the 
aggregate prediction, but involves their covariances. For example, if there 
are K predictions generated from a set of K, Xq*s, where 

^oi = « ^ ^Oi 



K K K 

= I VAR (Y ) + I I GOV (y^,, Y^ 1 for i M 
i=l ^ i«l j=l 

Since one cannot in general say whether the covariance terms are positive or 
negative, it is not possible to derive a simple expression, as done above, 





:^ee: Carl F. Christ, Econometric Models and Methods (New York: 
John Wiley & Sons, Inc., 1966, p. 551. 

The reason that the predictions are not independent is that all are 

A, /> 

generated from the parameters a and 8. An expression for the variance- 
covariance matrix for a set of predictions from a single regression may be 
found in Christ, op. cit., p. 551. 

See Christ, op> c it., p. 338. 
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for the relationship between the sample size and the variance of the ag- 
ftrcRHto prediction. 

A Further Note on Sample Design 

The problem of choosing a sample design is very complex and entails 
more than just presetting tolerance levels, guesstimating residual vari- 
ances and the signs of variance-covariance matrices. It involves careful 
consideration of the cost of the sample, and the relationship of the client 
population to the survey control group. Conlisk and Watts have developed a 
general way to construct an experimental design that is not only imaginative, 
but powerful and practical. While a full exposition of their approach is 
beyond the scope of this paper, an operational version of their model can set 
up an experiment design (specify a sampl") that will minimize the variance of 
any linear combination of predictions subject to a constraint on the costs of 
sampling. Further added weight can be placed in a subsample of particular 
interest. For example, one might be particularly interested in the behavior 
of the handicapped population and might thus be interested in minimizing the 
prediction variance for this subpopulation. 



See: John Conlisk and Harold Watts, "A Model for Optimizing Experimental 
Designs for Estimating Response Surfaces," in Harold W. Watts, John Conlisk, 
D. Lee Bawden and Larry L. Orr, eds.. Field Experimentation in Income Mainte- 
nance , Reprint 54, Institute for Research on Poverty, University of Wisconsin, 
1970, pp. 150-56. 
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VI. A NATIONAL LONGITUDINAL SURVEY 

The development of an evaluation-management information system for 
vocational rehabilitation requires an appropriate national labor market 
survey which measures economic performance over time, and which cap be 
used as the basis for the construction of a statistical control group. 
Unfortunately, one of the major failings of present longitudinal surveys 
are the methods they employ to collect labor market and employment infor- 
mation. For example, the Income Dynamics Panel of the University of 
Michigan's Survey Research Center uses traditional "last week" and "gross 

it 

period recall" questions. While such questions are appropriate for the 
Census's Current Population Survay and are useful in measuring the amount 
of employment and unemployment, they do not measure the time-phasing of 
employment and unemployment. As a result, they provide little help in 
attempts to measure detailed labor market behavior. 

Recently, several efforts have been made to collect more detailed 
continuous work histories. However, present attempts, most notably 
the National Longitudinal Survey, have been poorly structured and have not 
adequately dealt with secondary (moonlighting) employment and job search. 
This section will present an alternative continuous work history design, 
which it is hoped will more accurately record the respondent's labor market 
experience. 



An example of a "last week" question is, "What were you doing last 
week — working, looking for work, or something else?" 

An example of a "gross period recall" question is, "How many weeks 
did you work in the last 12 months?" 

For an example of the problems caused by a lack of time-phasing of 
unemployment in the Income Dynamics Panel see: F. W. Blackwell, D. H. Greenberg, 
A. J. Lipson, B. D. Rostker, S. T. Wolfberg, Performance Rewards for Services 
to the Employable Poor; A Proposed Incentive Pay System for California Job 
Agents, R-1028-HRD, The Rand Corporation, June 1972, p. 55. 
'k'k'k 

For a review of recent efforts by the Bureau of the Census see: 
Marie G. Argana, "Methods of Collecting Work History Information," in 
Evaluating the Impact of Manpower Programs , Michael E. Borus, ed. (Lexington, 
Massachusetts: D. C. Heath and Company, 1972, pp. 71-78). 
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A Design for a Continuous Work History Survey 

Most attempts to record continuous work histories have relied upon 
respondents to recall major periods of employment and unemployment. How- 
ever, letting respondent define recall periods runs the risk that the 
respondent will omit relevant periods of employment and unemployment, both 
In the primary and secondary labor markets. Argana found that when using re- 
spondent defined periods "some jobs were omitted from the work history 
because the respondent forgot about them or because the questionnaire 
skip patterns did not allow for the job to be recorded."* The following 
survey scheme is based upon standard recall periods of single calendar 
months. Calendar months are a universal standard which relate to major 
events (holidays), and provide a common structure fpr comparison. In this 
scheme each month is treated as a discrete time period. Events that carry 
over from month to month are noted by determining if a particular behavior, 
i.e., a given job, occurred throughout the month or only at the beginning 
or end of the month. 

For any given month the questioning starts by asking, "Were you employed 
during the month?" The exact flow and sequence of questions is described in 
Diagram I. The question flow is designed to account for each work day during 
the month and allows for single employment, multiple employment—both sequen- 
tial and concurrent, job search, training and withdrawals from the labor force. 
There are two general types of questions— flow regulators, indicated by a 
diamond, and descriptive, indicated by a box. Flow regulators are designed 
to direct the respondent to a further descriptive section. An example of a 
flow regulator is, "Were you employed at the beginning of the month?" A 
descriptive section is designed to elicit information from the respondent. 
There are five major descriptive sections. They are: Last Job, Job Descrip- 
tion, Job Search, Never Worked, Training, and Nonlabor Force Participation. 
The specific questions and responses for each of the major descriptive sec- 
tions are presented as Figures I-V in the Appendix. 

In the course of administering the questionnaire, and even within a given 
month, a respondent will be directed by a flow regulator to return to a major 
descriptive section several times. For example, in a given month, a respondent 
might be employed, fired, gone through a successful job search, then searched 



Argana, op. cit ., p. 75. 
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for a moonlighting job and found it. In such a case there would be three 
job descriptions (the first primary job, the second primary job, and the 
moonlighting job), and two job searches (the search for the second pri- 
mary job and the search for the moonlighting job). Throughout the 
questioning, probing questions acting as flow regulators, direct the 
respondent to return to relevant sections until all work days are ac- 
counted for. 

While the above continuous work history design has not been field 
tested, it is structured so that an accurate record of a respondent's 
labor market experience should be possible. 
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VI. SUMMARY AND CONCLUSION 

During the past fifteen years there has been a marked increase in the 
government's provision of vocational rehabilitation and manpower services • 
The government has actively attempted to increase the employability of 
mentally, physically and culturally handicapped persons by providing a 
variety of rehabilitative programs. Concurrent with the development of 
these services has been the extension of benefit-cost analysis into areas 
of human capital formation and as a tool to evaluate specific vocational- 
manpower programs. While benefit-cost analysis has been helpful in assessing 
the viability of the basic concept of rehabilitation, it has proven of little 
value to program and project managers who, faced with limited budgets, have 
to make tradeoff decisions among programs and projects. 

The failure of benefit-cost analysis to deal effectively with operational 
problems is seen by the distinction often made between process and outcome 
evaluation. Process evaluation is often defined as administrative monitoring. 
Outcome evaluation is defined as benefit-cost analysis. While evaluators 
continue to talk of the "net benefit" derived from an individual project, 
project managers continue to evaluate performance and make resource decisions 
on such process indicators as case load, training class size, and case closures. 
Such measures are not only meaningless in terms of the needs of the client pop- 
ulation, they establish perverse incentives and behavior which is inconsistent 
with program goals. It makes little sense to establish operating incentives 
that are not supportive of program objectives, or to evaluate a program when 
those charged with carrying it out respond to different incentives. 

There is a clear need for an evaluation-management information system 
which provides a systematic means for making inter-program and inter-project 
tradeoffs, and which establishes consistent incentives and standards to mea- 
sure internal project efficiency. If such a consistent evaluative system is 
to be built there must be agreement on an articulated set of goals which can 
be objectively related to accepted output measures. In the area of vocational- 
manpower programs there appears to be general agreement that the primary goal 
is the improvement in the employment performance of program participants, and 
that this can best be measured as an improvement in the client's income stream 
after receiving program services. 
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While there is general agreement on the primary goal of vocational- 
manpower programs, there is little agreement on how to treat such things 
as dependence and nonlabor market performance. This disagreement often 
manifests itself in the different ways benefit-cost studies treat secon- 
dary benefits and spillover effects. While the full development of a 
benefit-cost methodology is beyond the scope of this paper, agreement 
must be reached before a consistent evaluative system can be developed. 

A further problem which has prevented the establishment of a system 
to evaluate large-scale, continuing vocational-manpower programs is agree- 
ment on an appropriate control gtoup and a methodology to answer the ques- 
tion, "What would have been the client's income had he not participated in 
the program?" While there are several control groups which :ould be em- 
ployed, for a variety of reasons, the one best suited for the evaluation 
of large-scale, continuing manpower programs is one in which the client's 
foregone income is predicted, using a sample survey and r»-gression analysis. 
In such a procedure a regression equation, whose dependent variable is 
income, is fitted to the sample survey data. The equation explicitly con- 
trols for — the independent variables are — demographic factors, work history, 
local labor market factors, duration of current unemployment, factors re- 
lating to program participation and the availability of similar services to the 
survey population, and factors relating to the type and degree of the handi- 
cap. The attributes of individual clients are substituted into the fitted 
regression equation to predict the income the clients would have had, had 
they not participated in the program. The difference between this projection 
and actual income observed after completing the program is the measure of 
income gain and program performance. 

Criticai* to the above procedure is the variance of the predicted income. 
If income cannot be predicted within acceptable tolerances, then there is 
little reliable information on which to calculate income gain or to evaluate 
the program. In simple terms, the variance of the prediction is related to 
the goodness of fit of the original regression equation to the sample survey 
data, the distance in standard deviation units between the attributes of the 
client and the mean of the sample survey population, and the size of the 
original sample survey. If precision can be obtained in the prediction, i.e., 
the regression equation fits the sample survey reasonably well, the individual 
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client*s characteristics are relatively close to the mean of the sample 
population and the original sample survey is relatively large, then this 
procedure will provide reliable information on which to evaluate the pro- 
gram. 

The development of an evaluative-management information system re- 
quires an appropriate national labor-market survey which measures economic 
performance over time, and which can be used as the basis for the construc- 
tion of a statistical control group. Unfortunately, a major failing of 
present longitudinal surveys is the method they employ to collect labor 
market and employment data. In its final section the paper outlines an 
alternative to existing continuous work histories. While the design has 
not been field tested, it is structured so that an accurate record of a 
respondent's labor-market experience should be possible. 

In sum, the need for an evaluation-management information system to 
measure the effect of vocational-manpower programs is clear. Such a system 
must provide for inter-program and project tradeoffs, and must establish 
consistent Incentives and standards. Finally, a national sample survey, 
with a well-structured continuous work history questionnaire and regression 
analysis, should provide an adequate control group for the evaluation of 
large-scale, continuous manpower programs. 
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APPENDIX 



Figure I 
LAST JOB 



Questions 



I. When did you first work on your main last job? 

When did you leave your main last job? (If infor- 
mation was already collected as part of a previous 
month's response, do not repeat.) 

II. Was it your practice to have more than one job at 
a time? 

III. Your last job was in what type of industry? 

what type of occupation? 
main — normal function? 

IV. When you worked: 

How many hours per day did you normally work at 
this job? 

How many days per week did you normally work at 
this job? 

V. What time of day did you normally work? 



VI. How much did you normally earn? 



Why did you terminate from this job? 



Responses 

month, day, year 
month, day, year 



yes/no 

industrial code 
occupation code 
write in 



hours per day 
days per week 



varied/from (time) 
to (time) 



j)er hr. or 
day/week/ 



$_ 

$ 

biweekly/monthly/ 
yearly /other 

personal- family/ 
school/health/retirement/ 
seasonal job/slack work/ 
temporary job completed/ 
fired/quit/too far from 
home/poor working con- 
ditions/poor pay/to 
look for a better job 
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Figure II 
JOB DESCRIPTION 



Questions 



[. Did you work the entire month on this job? 
If no, day (within month) job started? 
If no, day job terminated. 



II. 



III. 



IV. 



Did you consider this: 
permanent or temporary? 
main or secondary? 

This job was in what type of industry? 

type of occupation? 
main (normal) function? 

When you worked: 

How many hours a day did you normally 
work at this job? 

H w many days per week did you normally 
work at this job? 

During this month did you work less than 
the normal work period? 
If yes, why? 



If yes, 

Did you average less hours per day? 
What was your average? 

Did you average less days per week? 
What was your average? 

Did you work less weeks per month? 
How many? 

During this month did you work more than the 
normal work period? 
If yes, why? 
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If yes, 

Did you average more hours per day? 
What was your average? 

Did you average more days per week? 
What was your average? O^S 



Responses 



yes/no 

day 

day 



permanent/ temporary 
main/secondary 

industrial code 
occupation code 
write in 



hours per day 
days per week 
yes/no 

slack work/holiday/ 
labor dispute/bad 
weather/own illness/ 
illness in family/ 
vacation/did not 
want to work normal 
period 



yes/no 

average hours 
yes/no 

average days 

yes/no 
weeks 

yes/no 

busy season/to fill 
in for another worker/ 
unusual increase in 
business/just wanted 
to work more 

yes/no 

average hours 
yes/no 

average days 
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JOB DESCRIPTIOr (CON^T) 
Questions 



V. What time of day did you normally work? 



VI. How much did you normally earn? 



VII. How much did you earn during this month? 

Was this less than you normally earned from 
this job? 

If yes, why? 

Was this more than you normally earned from 
this job? 

If yes, why? 



VIII* Did you terminate from this job during this month? 
If yes, why? 



Responses 



Varied/from (time) 
to (time) 

$ per hour or $ 

day/week/biweekly/ 
monthly /yearly/other 

$ 

yes/no 

did not work the normal 
period/docked pay 

yes/no 

bonus/raise during month/ 
overtime 

yes /no 

personal-family /school/ 
heal th/ re t iremen t / 
seasonal job/slack work/ 
temporary job completed/ 
fired/quit, too far from 
home/poor working con- 
ditions/poor pay/better 
job offer/to look for 
a better job 
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Figure III 
JOB SEARCH 

Questions 



I. Why were you looking for work? 



11. Were you searching for a job during the entire 
month? 

If no, day job search started? 
I' no, day job search ended? 

III. When you started looking for work what type of 
job were you looking for? 
permanent or temporary 
full-time or part-time 
industry 
occupation 
main function 
hours per day 
days per week 
time of day 

IV. When you first started looking for work what was 
the minimum wage you would have accepted? 



V. Did the following provide help in trying to find 
work? Rate and Rank. 

public employment service 
friends and relatives 
want ads /radio /TV 
private employment agencies 
unions 

former employers 
other 

VI. How many job interviews with prospective employers 
did you have during the period you were looking 
for work? 



Responses 

supplement income/ 
substitute for my lost 
job/substitute for my 
present job/substitute 
for lost job of other 
family member 



yes /no 

day 
day 



permanent /temporary 

full/part 

industry code 

occupation code 

write in 

hours 

days 

varied/start and stop time 



$ per hour or $ 

day / week/b iweekly / 
monthly /yearly /other 



Rate as to: 

used very often 
used sometimes 
used frequently 
never used 

Rate as to: 

quality of help 

1 = most helpful 



number 
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JOB SEARCH (CON'T) 



Questions 

VII. Did you receive any firm job offers which you 
did not take? 

If yes, describe the best job offer you did 
not take. 

permanent or temporary? 

full-tiiae or part-time? 

industry 

occupation 

main function 

hours per day 

days per week 

time of day 

earnings 



Why didn't you take this job? 



VIII. During the period did you change your mind about 
what minimum wage you would accept? 

What was your new minimum? 



When did you change your mind? 
Why did you change your mind? 



IX. How were you ablet to live — where did you get money 
to buy things when you were looking for work? 



X. Did your search for a job end during this month? 



If found work, why did you take this job? 
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Responses 
yes /no 



permanent/ temporary 

full/part 

industry code 

occupation code 

main function 

hours 

days 

varied/start and stop time 

$ per hour or $ 

day/week/biweekly/monthly/ 
yearly/other 

too far from home/ 

poor working conditions/ 

poor pay/poor job stability 

yes /no 

$ per hour or $ 

day /week/biweekly /monthly/ 
yearly/other 

day 

unemployed too long and 
needed uork/lost too 
many jobs because of 
too high a min/better 
nonwage compensation 

family/savings/welfare/ 
retirement income/other 
government payment/ 
charity/borrowed/ 
insurance/other 

found work/gave up and 
stopped looking/still 
looking 

was what I was looking for/ 
not quite right but needed 
work 
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Questions 



If gave up and stopped looking, what were the 
reasons that you were unable to find work? 



If gave up and stopped looking, why are you 
not looking now? 



Responses 

no job available/ 
employers think I 
am too young or too 
old/lack of skills/ 
lack of experience/ 
lack of education/ 
lack of references/ 
lack of tools/licenses, 
union membership/ 
language barrier/ 
physically handicapped/ 
police record/other 

entered training school/ 
cannot arrange child care/ 
change in personnel 
situation 
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Figure IV 
NEVER EMPLOYED 



Questions 

!• Have you ever looked for work? 

If yes, when did you first seek work? 

If yes, when did you last seek work? 

When you last looked for work, what 
was the reason that you were unable 
to find work? 



II. 



Since you left high school have you spent 
major portion of your time: 



in an institution? 
in a hospital? 
in the Armed Forces? 
ill at home? 
stayed at home? 
vocational school? 

III. How were you able to live — where did you get 
money to buy things?, 
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Responses 

yes /no 
month, year 
month, year 



no jobs available/ 
employers think I am 
too old or too young/ 
lack of skills/ 
lack of experience/ 
lack of education/ 
lack of references/ 
lack of tools/licenses/ 
union membership 
language barrier/ 
physically handicapped/ 
police record/other 



yes /no 
yes/no 
yes/no 
yes/no 
yes /no 
yes /no 



family /savings/welfare/ 
other government payment/ 
charity /borrowed /insurance/ 
other 
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Figure V 
TRAINING 



Questions 



I. What type of training were you in? 



II. Is this a government sponsored training program? 
If yes, which one? 

III. Were you in schoo]./ training the entire month? 

If no, start and stop days. 

If no, did you complete the program? 

If no, why did you not complete the 

program? 



Responses 



regular high school/ 
college/vocational 
high school/vocational 
rehabilitation/business 
school/technical insti- 
tute/OJT/work experience 
program 

yes/no 

WIN/CEP/MDTA/Job Corps/ 
OIC/NYC/JOBS/other 

yes /no 

days 
yes/no 

currently enrolled/found 
job/too much time involved/ 
lost interest/too difficult 
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Figure VI 
NONLABOR FORCE PARTICIPATION 

Questions 

I* Were you out of the labor force the entire month? 
If no, start and stop days? 

II. What were you doing when you were not in the 
labor force? 



III. Why were you not in the labor force? 



IV. How were you able to live — where did you get 
money to buy things when you were not in the 
labor force? 



Responses 

yes/no 
days 

in institution/ 

in hospital/in Armed 

Forces/illness at home/ 

stayed at home/vacation/ 

school 

illness/in institution/ 
in Araed Forces/ 
student /vacation/ 
did not want to work/ 
child care 

family/ savings/welfare/ 

retirement income/ 

other government payment/ 

cha r i t y / bor rowed / ins ur a nc e / 

other 
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